Skip to content

Add experimental skill/<name> user-agent pairs from DATABRICKS_SKILLS_INVOKED - #7730

Merged
chrisst merged 11 commits into
mainfrom
chris.stephens/skills-invoked
Nov 18, 2026
Merged

chrisst merged 11 commits into
mainfrom
chris.stephens/skills-invoked

Conversation

@chrisst

@chrisst chrisst commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Experimental skill-usage tracking. When DATABRICKS_SKILLS_INVOKED=databricks-core,databricks-jobs is set, the CLI adds one skill/<name> pair per skill to its user agent: ... skill/databricks-core skill/databricks-jobs. Databricks agent skills will prefix their CLI calls with this env var (databricks/databricks-agent-skills#311).

  • Env var only, no flag. It can be removed later without changing the CLI's interface. Older CLIs also ignore unknown env vars, so skills need no version gate.
  • Respects the telemetry opt-out. Nothing is added when DATABRICKS_CLI_DISABLE_TELEMETRY is set.
  • Normalization. Entries are trimmed, lowercased and deduplicated. Entries not matching ^[a-z0-9][a-z0-9_.-]*$ (a subset of what the SDK allows in a user agent value) or longer than 64 bytes are dropped. Names are kept in order while all skill/<name> pairs fit in 225 bytes (five of the longest current skill names), so the SDK's auth/, agent/ and other pairs stay within the 512-character usage-log cap. Bad input never fails or warns a command; dropped entries are only logged at debug level.
  • Placement. It lives in cmd/root/user_agent_skills.go next to the other user-agent dimensions and is applied in the root PersistentPreRunE. Every request that already carries cmd/ carries the skills too.

The DECO usage pipeline (deco/bundles/http_access_logs → deco_usage_logs_normalized) parses repeated keys into unified_user_agent.other_info_multi, as it already does for the repeated aitools/ pairs. So the skills land in other_info_multi['skill'] with no pipeline change.

Known gap: the files/v2 client used by fs on /Volumes builds its own user agent without any of the CLI's context pairs (no cli/ or cmd/ either), so those requests carry no skills.

Why

To measure which Databricks agent skills drive CLI usage, per request, from the existing HTTP access logs, with no new header or log column.

Tests

  • Unit (cmd/root_test): parsing table (normalization, disallowed characters including :, name length, and 225-byte budget boundaries), and the user-agent pairs with and without the telemetry opt-out. Both env vars are always set in the context so a host value can't leak in.
  • Acceptance (acceptance/cmd/skills-invoked): asserts skill/ pairs in the recorded user agent for current-user me, api get, and bundle validate. It also covers normalization, no pairs when unset, and no pairs with telemetry disabled. A host DATABRICKS_SKILLS_INVOKED is unset in acceptance_test.go.

This pull request and its description were written by Isaac.

…KILLS_INVOKED

Agent skills that drive a CLI call can list themselves in the
DATABRICKS_SKILLS_INVOKED environment variable (or a hidden
--skills-invoked flag, which wins). The CLI normalizes the list and sends
it as X-Databricks-Skills-Invoked on Databricks API requests. Invalid
entries are dropped silently, with debug logging only.

The value is resolved once in the root PersistentPreRunE and stored in the
command context. Every SDK config site applies a header hook that chains any
existing hook, so liteswap keeps working. The files client gets the header
through a wrapping transport.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the DABs DABs related issues label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Commit: b1c946be4 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (e4e790c) latest (v1.26.0)
TestAccept/cmd/skills-invoked ✅ ❌ ❌
TestAccept/cmd/skills-invoked: branch ✅ | main (e4e790c) ❌ | latest (v1.26.0) ❌

main (e4e790c):

acceptance_test.go:1388: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1388
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1129
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:674
        	Error:      	Not equal: 
        	            	expected: "\n=== Env var adds the skills to the user agent on a workspace GET\n\n>>> [CLI] current-user me\n\n>>> print_requests.py --get //scim/v2/Me\n\n=== Env var adds the skills on databricks api\n\n>>> [CLI] api get /api/2.1/clusters/list\n\n>>> print_requests.py --get //clusters/list\n\n=== Env var adds the skills from bundle commands\n\n>>> [CLI] bundle validate\n\n>>> print_requests.py --get //scim/v2/Me\n\n=== Invalid entries are dropped and the rest is normalized\n\n>>> [CLI] current-user me\n\n>>> print_requests.py --get //scim/v2/Me\n\n=== No skills when unset\n\n>>> [CLI] current-user me\n\n>>> print_requests.py --get //scim/v2/Me\n\n=== No skills when telemetry is disabled\n\n>>> [CLI] current-user me\n\n>>> print_requests.py --get //scim/v2/Me\n"
        	            	actual  : "\n=== Env var adds the skills to the user agent on a workspace GET\n\n>>> [CLI] current-user me\n\n>>> print_requests.py --get //scim/v2/Me\ncontains error: ' skill/databricks-core skill/databricks-jobs' not found in the output.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -6,32 +6,5 @@
        	            	 >>> print_requests.py --get //scim/v2/Me
        	            	+contains error: ' skill/databricks-core skill/databricks-jobs' not found in the output.
        	            	 
        	            	-=== Env var adds the skills on databricks api
        	            	+Exit code: 1
        	            	 
        	            	->>> [CLI] api get /api/2.1/clusters/list
        	            	-
        	            	->>> print_requests.py --get //clusters/list
        	            	-
        	            	-=== Env var adds the skills from bundle commands
        	            	-
        	            	->>> [CLI] bundle validate
        	            	-
        	            	->>> print_requests.py --get //scim/v2/Me
        	            	-
        	            	-=== Invalid entries are dropped and the rest is normalized
        	            	-
        	            	->>> [CLI] current-user me
        	            	-
        	            	->>> print_requests.py --get //scim/v2/Me
        	            	-
        	            	-=== No skills when unset
        	            	-
        	            	->>> [CLI] current-user me
        	            	-
        	            	->>> print_requests.py --get //scim/v2/Me
        	            	-
        	            	-=== No skills when telemetry is disabled
        	            	-
        	            	->>> [CLI] current-user me
        	            	-
        	            	->>> print_requests.py --get //scim/v2/Me
        	            	-
        	Test:       	TestAccept/cmd/skills-invoked
        	Messages:   	cmd/skills-invoked/output.txt vs /tmp/TestAcceptcmdskills-invoked501370321/002/output.txt
    acceptance_test.go:1166: 
        LOG.me: {
        LOG.me:   "headers": {
        LOG.me:     "User-Agent": [
        LOG.me:       "cli/1.27.0-dev databricks-sdk-go/0.201.0 go/1.27.4 os/linux cmd/current-user_me cmd-exec-id/b938793f-db82-4d05-999d-c498ef409d2e interactive/none auth/pat cicd/github"
        LOG.me:     ]
        LOG.me:   },
        LOG.me:   "method": "GET",
        LOG.me:   "path": "/api/2.0/preview/scim/v2/Me"
        LOG.me: }
FAIL

latest (v1.26.0):

acceptance_test.go:1388: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1388
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1129
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:674
        	Error:      	Not equal: 
        	            	expected: "\n=== Env var adds the skills to the user agent on a workspace GET\n\n>>> [CLI] current-user me\n\n>>> print_requests.py --get //scim/v2/Me\n\n=== Env var adds the skills on databricks api\n\n>>> [CLI] api get /api/2.1/clusters/list\n\n>>> print_requests.py --get //clusters/list\n\n=== Env var adds the skills from bundle commands\n\n>>> [CLI] bundle validate\n\n>>> print_requests.py --get //scim/v2/Me\n\n=== Invalid entries are dropped and the rest is normalized\n\n>>> [CLI] current-user me\n\n>>> print_requests.py --get //scim/v2/Me\n\n=== No skills when unset\n\n>>> [CLI] current-user me\n\n>>> print_requests.py --get //scim/v2/Me\n\n=== No skills when telemetry is disabled\n\n>>> [CLI] current-user me\n\n>>> print_requests.py --get //scim/v2/Me\n"
        	            	actual  : "\n=== Env var adds the skills to the user agent on a workspace GET\n\n>>> [CLI] current-user me\n\n>>> print_requests.py --get //scim/v2/Me\ncontains error: ' skill/databricks-core skill/databricks-jobs' not found in the output.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -6,32 +6,5 @@
        	            	 >>> print_requests.py --get //scim/v2/Me
        	            	+contains error: ' skill/databricks-core skill/databricks-jobs' not found in the output.
        	            	 
        	            	-=== Env var adds the skills on databricks api
        	            	+Exit code: 1
        	            	 
        	            	->>> [CLI] api get /api/2.1/clusters/list
        	            	-
        	            	->>> print_requests.py --get //clusters/list
        	            	-
        	            	-=== Env var adds the skills from bundle commands
        	            	-
        	            	->>> [CLI] bundle validate
        	            	-
        	            	->>> print_requests.py --get //scim/v2/Me
        	            	-
        	            	-=== Invalid entries are dropped and the rest is normalized
        	            	-
        	            	->>> [CLI] current-user me
        	            	-
        	            	->>> print_requests.py --get //scim/v2/Me
        	            	-
        	            	-=== No skills when unset
        	            	-
        	            	->>> [CLI] current-user me
        	            	-
        	            	->>> print_requests.py --get //scim/v2/Me
        	            	-
        	            	-=== No skills when telemetry is disabled
        	            	-
        	            	->>> [CLI] current-user me
        	            	-
        	            	->>> print_requests.py --get //scim/v2/Me
        	            	-
        	Test:       	TestAccept/cmd/skills-invoked
        	Messages:   	cmd/skills-invoked/output.txt vs /tmp/TestAcceptcmdskills-invoked2668204533/002/output.txt
    acceptance_test.go:1166: 
        LOG.me: {
        LOG.me:   "headers": {
        LOG.me:     "User-Agent": [
        LOG.me:       "cli/1.26.0 databricks-sdk-go/0.197.0 go/1.27.4 os/linux cmd/current-user_me cmd-exec-id/6d939ce6-f29d-4e11-a83d-faa7d31ba48b interactive/none auth/pat cicd/github"
        LOG.me:     ]
        LOG.me:   },
        LOG.me:   "method": "GET",
        LOG.me:   "path": "/api/2.0/preview/scim/v2/Me"
        LOG.me: }
FAIL

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: b1c946b

Run: 44327434044

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 20 17:18
💚​ aws-windows-latest-4core-16gb 1 274 18 12:26
💚​ azure linux-2core-8gb 1 271 20 14:02
💚​ azure-windows-latest-4core-16gb 1 273 18 11:20
💚​ gcp linux-2core-8gb 1 272 20 23:08
💚​ gcp-windows-latest-4core-16gb 1 274 18 9:50
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 48 slowest tests (at least 2 minutes):
duration env testname
6:03 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
5:05 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
5:01 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
4:50 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
4:18 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
4:11 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
3:58 gcp linux-2core-8gb TestImportDirWithOverwriteFlag
3:47 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
3:36 aws-windows-latest-4core-16gb TestFilerWorkspaceNotebook/pythonJupyterNb.ipynb
3:33 aws linux-2core-8gb TestImportDirWithOverwriteFlag
3:18 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
3:16 azure-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
3:13 azure-windows-latest-4core-16gb TestAccept
3:07 gcp linux-2core-8gb TestImportDirDoesNotOverwrite
3:07 aws-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
3:05 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:58 gcp-windows-latest-4core-16gb TestAccept
2:55 aws-windows-latest-4core-16gb TestLockUnlockWithAllowsLockFileNotExist
2:53 azure-windows-latest-4core-16gb TestImportDir
2:50 aws linux-2core-8gb TestSyncIncrementalFileOverwritesFolder
2:50 aws-windows-latest-4core-16gb TestFilerReadDir/workspace_files
2:50 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
2:43 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
2:43 aws-windows-latest-4core-16gb TestSyncNestedFolderDoesntFailOnNonEmptyDirectory
2:40 aws-windows-latest-4core-16gb TestAccept
2:35 azure-windows-latest-4core-16gb TestSyncIncrementalFileSync
2:35 gcp linux-2core-8gb TestFilerReadWrite/workspace_files_extensions
2:34 gcp linux-2core-8gb TestSyncIncrementalSyncPythonNotebookToFile
2:33 azure linux-2core-8gb TestImportDirDoesNotOverwrite
2:30 gcp linux-2core-8gb TestSyncFullFileSync
2:30 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
2:30 azure linux-2core-8gb TestFilerRecursiveDelete/workspace_files
2:30 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
2:28 aws linux-2core-8gb TestLock
2:28 gcp linux-2core-8gb TestSyncNestedSpacePlusAndHashAreEscapedSync
2:26 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:25 azure-windows-latest-4core-16gb TestFilerWorkspaceNotebook/pyNb.py
2:23 azure linux-2core-8gb TestImportDir
2:19 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
2:18 aws-windows-latest-4core-16gb TestFilerReadWrite/workspace_files
2:18 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsRead
2:15 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
2:13 gcp-windows-latest-4core-16gb TestExportDir
2:05 azure-windows-latest-4core-16gb TestWorkspaceFilesExtensions_ExportFormatIsPreserved/jupyter_r
2:04 aws-windows-latest-4core-16gb TestWorkspaceFilesExtensionsNotebooksAreNotReadAsFiles
2:01 gcp-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
2:00 aws linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions
2:00 gcp linux-2core-8gb TestSyncIncrementalFileSync

Every SDK config site applies liteswap before skillsinvoked, and
skillsinvoked.Apply chains the existing header hook, so liteswap does not
need to change. Shorten doc comments to the contract and the reasons.

Co-authored-by: Isaac <no-reply@databricks.com>
@chrisst
chrisst marked this pull request as ready for review November 18, 2026 10:56
@chrisst
chrisst requested review from a team as code owners November 18, 2026 10:56

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

The HTTP access log truncates the header at 1024 bytes. Stop adding names
before the joined value would exceed that, so a partial name is never logged.

Co-authored-by: Isaac <no-reply@databricks.com>

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at 954546c. Each finding was verified against the code; simonfaltum may follow up.

Not approving yet: 1 blocking issue(s), tagged [blocking]; CI failures caused by this PR: task test (windows, direct), task test (windows, directdms), test-result.

This PR sends the agent skills that drove a CLI call as X-Databricks-Skills-Invoked, read from DATABRICKS_SKILLS_INVOKED or a hidden flag, and attaches it to the SDK clients, bundles, databricks api, and the files/v2 client. The parsing, the size caps, the chaining onto liteswap's hook, and the request-cloning transport all look correct. The new acceptance test fails on both Windows CI jobs because Git Bash rewrites the /api/2.0/... argument, so this needs one MSYS_NO_PATHCONV=1 fix before merging. The rest are small coverage and test-hygiene suggestions, plus a question about whether the hidden flag is needed.

Questions

  1. Is the hidden --skills-invoked flag needed? The comment in cmd/root/skills_invoked.go says skills set DATABRICKS_SKILLS_INVOKED because older CLIs reject unknown flags, so nothing in this PR uses the flag. It still adds a root persistent flag to every command, along with precedence logic and tests. Invalid values are dropped without a message, which AGENTS.md discourages for flags ("Never silently ignore a flag…"). If there is no planned user, would you drop the flag and keep only the env var? If there is one, could you name it in the description? (non-blocking)

Accepted answers

  • Skills header depends on liteswap being applied first
  • Invalid skill names are dropped silently

Comment thread acceptance/cmd/skills-invoked/script Outdated
trace print_requests.py --get //clusters/list

title "Env var sends the header on databricks api"
DATABRICKS_SKILLS_INVOKED=databricks-core trace $CLI api get /api/2.0/clusters/list > /dev/null

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] On Windows, Git Bash rewrites the leading-slash argument /api/2.0/clusters/list into /Program Files/Git/api/2.0/clusters/list. The testserver then 404s, api get exits 1, and the rest of the script never runs. Both Windows task test jobs fail on TestAccept/cmd/skills-invoked with No stub found for pattern: GET /Program Files/Git/api/2.0/clusters/list. Prefix only this command with MSYS_NO_PATHCONV=1, as acceptance/cmd/api/*/script does: DATABRICKS_SKILLS_INVOKED=databricks-core MSYS_NO_PATHCONV=1 trace $CLI api get .... Don't set it test-wide in test.toml. The script runs print_requests.py, and .agents/rules/testing.md explains that the helper needs path conversion to work on Windows.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5de7d1b: the api get line now has MSYS_NO_PATHCONV=1 on that command only. The rest of the script still runs print_requests.py with path conversion on.

Comment thread libs/skillsinvoked/skills.go Outdated
// Apply sets the header on every request made with cfg, keeping any header
// hook already on cfg (such as liteswap's). The value comes from the request
// context, so it is only sent when the command context reaches the request.
func Apply(cfg *config.Config) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nice to have] databricks auth profiles builds its own config.Config in profileMetadata.Load (cmd/auth/profiles.go:48) and never calls Apply. Its CurrentUser.Me and Workspaces.List validation calls run on the command context but go out without the header. The SSH release-upload client has the same gap: newHTTP11WorkspaceClient (experimental/ssh/internal/client/releases.go:61) rebuilds the config from ConfigAttributes, which drops cfg.Headers. Skills often run auth profiles to check auth, and neither path is in the PR's "Known gaps". Either call skillsinvoked.Apply(cfg) in profileMetadata.Load and copy src.Headers in newHTTP11WorkspaceClient, or list both under "Known gaps".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5de7d1b: profileMetadata.Load now calls skillsinvoked.Apply(cfg), and newHTTP11WorkspaceClient copies src.Headers, since it's a func and the attribute loop skips it.

Comment thread libs/filer/files_client.go Outdated
client.WithoutConfigFile(),
client.WithoutEnv(),
// files/v2 doesn't use cfg.Headers. http.DefaultTransport is its default.
client.WithHTTPClient(&http.Client{Transport: skillsinvoked.Transport(http.DefaultTransport)}),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nice to have] The transport added here sets only the skills header, so other cfg.Headers hooks still don't reach the files/v2 client. For example, with DATABRICKS_LITESWAP_TARGET set, fs ls dbfs:/Volumes/... sends no x-databricks-traffic-id, while the other requests in the same command do. A transport in libs/filer that clones the request and calls cfg.Headers (when it is set) would send both headers. skillsinvoked.Transport would then not be needed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5de7d1b: libs/filer now has a headersTransport that clones the request and runs cfg.Headers when it's set, so liteswap's x-databricks-traffic-id reaches files/v2 too. skillsinvoked.Transport is gone. Unit tests cover the no-hook, hook and hook-error cases.

Comment thread cmd/root/skills_invoked.go Outdated
// the environment variable.
func withSkillsInvoked(ctx context.Context, cmd *cobra.Command) context.Context {
raw := env.Get(ctx, envSkillsInvoked)
if v := cmd.Flag(flagSkillsInvoked).Value.String(); v != "" {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The check uses Value.String() != "" instead of whether the flag was set. With DATABRICKS_SKILLS_INVOKED=from-env, --skills-invoked= still sends from-env, while --skills-invoked='!!' sends nothing (the unit test asserts that case). So the flag wins over the env var only when its value is non-empty. Use cmd.Flag(flagSkillsInvoked).Changed and add a test case for an explicit empty value. This goes away if the flag is dropped.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This went away with the flag: in 5de7d1b the CLI reads only DATABRICKS_SKILLS_INVOKED.

Comment thread cmd/root/skills_invoked_test.go Outdated
Comment on lines +33 to +35
if tt.env != "" {
ctx = env.Set(ctx, envSkillsInvoked, tt.env)
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] When tt.env is empty, the test doesn't override the variable, so env.Get falls back to os.LookupEnv. Running go test ./cmd/root from a shell where DATABRICKS_SKILLS_INVOKED is exported makes the neither case fail. The acceptance test has the same exposure: acceptance/acceptance_test.go unsets the host's agent env vars (including CLAUDE_CODE_CHILD_SESSION) but not this one. With it exported, "No header when unset" and the header-less fs mkdir PUT pick up the host value. Always call env.Set(ctx, envSkillsInvoked, tt.env): an empty value in the context overrides the OS environment. Also add DATABRICKS_SKILLS_INVOKED to the unset list in acceptance_test.go.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5de7d1b: the unit test always calls env.Set for both env vars, and acceptance_test.go now unsets DATABRICKS_SKILLS_INVOKED next to CLAUDE_CODE_CHILD_SESSION.

Comment thread acceptance/cmd/skills-invoked/test.toml Outdated
Comment on lines +5 to +7
[[Server]]
Pattern = "GET /api/2.0/clusters/list"
Response.Body = '{}'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This stub exists only because the script calls /api/2.0/clusters/list, and libs/testserver already models GET /api/2.1/clusters/list. .agents/rules/testing.md reserves [[Server]] for routes the testserver doesn't model. Call api get /api/2.1/clusters/list in the script, with the MSYS_NO_PATHCONV=1 fix, and drop the stub.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5de7d1b: the script calls api get /api/2.1/clusters/list and the [[Server]] stub is gone.

chrisst and others added 3 commits November 18, 2026 12:33
Only DATABRICKS_SKILLS_INVOKED is read now, so the feature can be removed
later without changing the CLI's interface. No header is sent when
DATABRICKS_CLI_DISABLE_TELEMETRY is set.

Review fixes:
- Acceptance test: MSYS_NO_PATHCONV=1 for the leading-slash api get so it
  passes on Windows, call the modeled /api/2.1/clusters/list and drop the
  stub, and unset a host DATABRICKS_SKILLS_INVOKED in acceptance_test.go.
- Unit test always sets the env vars in the context so a host value can't leak in.
- `auth profiles` validation calls and the SSH HTTP/1.1 release client now
  send the header.
- The files/v2 client applies the whole cfg.Headers hook in its transport,
  so liteswap's traffic header reaches it too; skillsinvoked.Transport is gone.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@chrisst
chrisst requested a review from a team as a code owner November 18, 2026 12:35
@chrisst chrisst changed the title Send X-Databricks-Skills-Invoked header from DATABRICKS_SKILLS_INVOKED / --skills-invoked Send X-Databricks-Skills-Invoked header from DATABRICKS_SKILLS_INVOKED Nov 18, 2026
@github-actions github-actions Bot added the auth label Nov 18, 2026
@chrisst

chrisst commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@simonfaltum thanks for the review. Everything is addressed as of 6268ba7:

  • Hidden flag: dropped. The CLI reads only DATABRICKS_SKILLS_INVOKED, so this can be removed later without changing the CLI's interface. It also sends nothing when DATABRICKS_CLI_DISABLE_TELEMETRY is set.
  • Windows acceptance failure: fixed with MSYS_NO_PATHCONV=1 on that one command, and the stub is gone.
  • Coverage: auth profiles and the SSH release client now send the header. The files/v2 transport applies the whole cfg.Headers hook, so liteswap reaches it too.
  • Test hygiene: both test layers now isolate the env var from the host.

I also merged main into the branch. Could you take another look?

chrisst added a commit to databricks/databricks-agent-skills that referenced this pull request Nov 18, 2026
databricks/cli#7730 now reads only DATABRICKS_SKILLS_INVOKED and sends
nothing when DATABRICKS_CLI_DISABLE_TELEMETRY is set.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Chris Stephens <chris.stephens@databricks.com>
DATABRICKS_SKILLS_INVOKED now adds one skill/<name> pair per skill to the
user agent, next to the other user-agent dimensions in cmd/root. The SDK
already sends context user-agent pairs on every request, so the per-client
header hooks (auth, bundles, api, auth profiles, SSH releases, files/v2) are
reverted. The DECO usage pipeline parses repeated keys into
unified_user_agent.other_info_multi, as it does for aitools/.

Names are limited to [a-z0-9_.-], a subset of what the SDK accepts in a user
agent value. This is experimental tracking and is commented as such.

Co-authored-by: Isaac <no-reply@databricks.com>
@chrisst chrisst changed the title Send X-Databricks-Skills-Invoked header from DATABRICKS_SKILLS_INVOKED Add experimental skill/<name> user-agent pairs from DATABRICKS_SKILLS_INVOKED Nov 18, 2026
@chrisst

chrisst commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@simonfaltum heads-up, the design changed after my last comment: skills now go into the user agent as skill/<name> pairs instead of a separate X-Databricks-Skills-Invoked header (395a3c9).

  • Why: the DECO usage pipeline already parses repeated user-agent keys into other_info_multi, so this needs no new header, no new log column and no per-client hooks.
  • What's left: cmd/root/user_agent_skills.go plus tests. The Apply calls, the files/v2 transport, and the auth profiles and SSH changes are reverted, since the user agent already covers those paths. That means your liteswap-on-files/v2 suggestion is no longer part of this PR; it would be a separate fix.
  • Kept from your review: env var only, the telemetry opt-out, the Windows fix and test isolation.

The PR is now 217 lines across 9 files. Could you take another look?

Co-authored-by: Isaac <no-reply@databricks.com>

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 704ed07f7f872f4a3d704e29ef8d5a9b5ede01a7 with the pr-review workflow; all five reviewers completed, and each reported finding was checked against the code.

Requesting changes: one blocking issue, one nice-to-have test fix, and four minor nits, detailed inline.

The user-agent approach, repeated skill/ pairs, bare-name contract, and telemetry opt-out look consistent with the intended design. The remaining blocker is the aggregate length: supported input can push existing auth and agent attribution beyond the usage-log user-agent limit. There is also a host-environment dependency in the acceptance fixture.

CI is green on this head, including all six Linux/macOS/Windows direct/directdms test jobs and Integration Tests; some other jobs were skipped. The earlier Windows path-conversion issue is fixed. The removal of the hidden flag and the documented files/v2 scope are accepted.

The length finding was verified through source tracing and a static calculation against the 512-character usage-log tag cap. No live ingestion reproduction or full local Go test run was performed.


This review was generated with GitHub MCP.

Comment thread cmd/root/user_agent_skills.go Outdated
const skillUserAgentKey = "skill"

const (
maxSkills = 20

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] Twenty accepted 64-character names produce 1,419 characters of skill/ pairs alone. The usage-log path feeding deco_usage_logs_normalized caps userAgent at 512 characters, and SDK v0.200.0 appends auth/, agent/, and other dimensions after these skills. This input therefore loses later skills and existing attribution metadata before the analytics parser sees it.

Please add an aggregate serialized-size budget with room for SDK metadata, or increase the receiving limit before enabling this, with boundary coverage. The 512-character cap is specific to usage logging; this was verified from source and a static calculation, without a live ingestion test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d6888e2: the 20-skill cap is now a 128-byte budget for all skill/<name> pairs together (maxSkillsUserAgentLength). Names are kept in order while they fit, so the auth/, agent/ and other SDK pairs still have room under the 512-character usage-log cap. About five typical skill names fit. Boundary tests cover fits, overflow, and exactly 128 bytes, and every case asserts the result stays within the budget.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up in b1c946b: I raised the budget to 225 bytes. 128 fit only two of the longest current skill names; 225 fits any five of today's 38 skills (the longest pair is 45 bytes, so five of those is 225). Without skills, the CLI's user agent is about 200 characters, so the total stays under the 512-character usage-log cap with room to spare.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chrisst could you pick up a follow-up fix for this user-agent length guard? The merged code still uses the 225-byte budget. I checked the merge: the relevant CLI code is unchanged from b1c946b, and the SDK upgrade from 0.200.0 to 0.201.0 doesn't change user-agent construction.

A local reproduction using the actual CLI skills helpers and SDK request visitors produces a 514-byte user agent for bundle deploy with an existing direct-engine lineage, Claude Code as caller, PAT auth, and Databricks plugins for Claude Code 0.2.9, Codex 0.2.8, and Copilot 0.2.9. These six existing skill names are all accepted and occupy exactly 225 bytes including the skill/ prefixes:

databricks-serverless-migration-api,databricks-lakeflow-connect-saas,databricks-serverless-migration,databricks-synthetic-data-gen,databricks-mlflow-evaluation,databricks-execution-compute

The other fields total 289 bytes, so the 512-character logging prefix cuts agent/claude-code to agent/claude-co. The previous 128-byte budget produced a 408-byte header for the same input and preserved the agent. This verifies header construction and the truncation boundary; it isn't a live ingestion test.

Could you restore the conservative budget for now, or account for the complete request user agent when choosing the skill allowance, and add a regression test covering bundle engine/lineage plus SDK auth/agent fields? The current boundary tests check only the skill portion.


This comment was generated with GitHub MCP.

@@ -0,0 +1,3 @@
RecordRequests = true

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nice to have] With DATABRICKS_CLI_DISABLE_TELEMETRY=1 exported in the invoking shell, this fixture inherits the opt-out through auth.ProcessEnv. The CLI correctly omits the skills, so the first positive contains.py assertion fails. Set Env.DATABRICKS_CLI_DISABLE_TELEMETRY = "" in this fixture; the final scenario can still set it to 1 inline.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d6888e2: the fixture sets Env.DATABRICKS_CLI_DISABLE_TELEMETRY = "", and the last scenario still sets it to 1 inline. I checked it passes with the opt-out exported in my shell.

Comment thread acceptance/cmd/skills-invoked/script Outdated

title "Env var adds the skills to the user agent on a workspace GET"
DATABRICKS_SKILLS_INVOKED=databricks-core,databricks-jobs trace $CLI current-user me > /dev/null
trace print_requests.py --get //scim/v2/Me | contains.py " skill/databricks-core skill/databricks-jobs" > /dev/null

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Keep the request dump in a LOG.<name> file before asserting it. The current redirection discards the actual user agent, and print_requests.py deletes the recording, so a missing-skill failure has little diagnostic output. .agents/rules/testing.md:209 asks for LOG.* instead of /dev/null.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d6888e2: each request dump goes to a LOG.<name> file, and contains.py reads from it, so a failed assertion keeps the actual user agent.

Comment thread cmd/root/user_agent_skills_test.go Outdated
@@ -0,0 +1,72 @@
package root

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] .agents/rules/testing.md:31 asks new tests to use the _test package suffix. Could these use root_test, with test-only aliases in the existing export_test.go if direct helper access is needed? Existing sibling user-agent tests also use package root, so this is only a nit against the written rule.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d6888e2: the tests are in package root_test, with aliases in export_test.go.

Comment thread cmd/root/user_agent_skills.go Outdated
"github.com/databricks/databricks-sdk-go/useragent"
)

// Experimental skill-usage tracking: agent skills prefix CLI calls with this

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Add a public source link for the skills producer or its environment-variable contract to this comment. .agents/rules/comments.md:62 requires a reference URL when integrating external tools or detecting their environment variables, and it would make the bare-name format easy to trace.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d6888e2: the comment links the databricks-core skill that tells agents to set the env var.

Comment thread cmd/root/user_agent_skills_test.go Outdated
{"trims and lowercases", " Databricks-Core , JOBS ", []string{"databricks-core", "jobs"}},
{"drops empty entries", ",,a,,", []string{"a"}},
{"allows underscore and dot", "a_b.c-d", []string{"a_b.c-d"}},
{"drops characters the user agent rejects", "ok,bad name,a:b,a/b,a+b", []string{"ok"}},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Rename this case to refer to disallowed skill-name characters. The input includes a+b, which the pinned SDK accepts in a user-agent value; the CLI intentionally uses a stricter pattern.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed in d6888e2 to "drops disallowed skill-name characters".

The usage logs that feed skill analytics cap the user agent at 512
characters, and the SDK appends auth/, agent/ and other pairs after the
skills. Replace the 20-skill cap with a 128-byte budget for all
" skill/<name>" pairs so those pairs survive, with boundary tests.

Also:
- Link the databricks-core skill that sets DATABRICKS_SKILLS_INVOKED.
- Move the unit tests to package root_test via export_test.go, and rename
  the disallowed-characters case.
- Acceptance: pin DATABRICKS_CLI_DISABLE_TELEMETRY empty so a host opt-out
  can't hide the skills, and keep request dumps in LOG.* files.

Co-authored-by: Isaac <no-reply@databricks.com>
@chrisst

chrisst commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@simonfaltum all six points from your latest review are addressed in d6888e2. The blocker is fixed with a 128-byte budget for the skill pairs, so the other user-agent pairs keep room under the 512-character usage-log cap. The fixture, LOG files, package, link and test name are done too; details are in each thread. Could you take another look?

chrisst and others added 2 commits November 18, 2026 14:20
Co-authored-by: Isaac <no-reply@databricks.com>
128 bytes fit only two of the longest current skill names. 225 bytes fits
five of the longest (45-byte " skill/<name>" pairs) and still leaves room
under the 512-character usage-log cap for the rest of the CLI's user agent,
which is about 200 characters.

Co-authored-by: Isaac <no-reply@databricks.com>
@chrisst
chrisst enabled auto-merge November 18, 2026 14:45
@chrisst
chrisst added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 62c8308 Nov 18, 2026
27 checks passed
@chrisst
chrisst deleted the chris.stephens/skills-invoked branch November 18, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auth DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants